%pip install torch
%pip install sentence_transformers
%pip install --upgrade pip
%pip install numpy pandas scikit-learn umap-learn cython
%pip install hdbscan --no-build-isolation
%pip install bertopic
%pip install vaderSentiment
%pip install accelerate
%pip install transformers==4.49.0
%pip install keybert sentence-transformers
%pip install bertopic[all]
%pip install wheel
%pip install cython
%pip install numpy
%pip install hdbscan
%pip install nltk
%pip install stanza

Requirement already satisfied: torch in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (2.7.0)
Requirement already satisfied: filelock in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from torch) (3.18.0)
Requirement already satisfied: typing-extensions>=4.10.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from torch) (4.12.2)
Requirement already satisfied: sympy>=1.13.3 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from torch) (1.14.0)
Requirement already satisfied: networkx in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from torch) (3.4.2)
Requirement already satisfied: jinja2 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from torch) (3.1.6)
Requirement already satisfied: fsspec in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from torch) (2025.3.2)
Requirement already satisfied: setuptools in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from torch) (78.1.0)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from sympy>=1.13.3->torch) (1.3.0)
Requirement already satisfied: MarkupSafe>=2.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from jinja2->torch) (3.0.2)
Note: you may need to restart the kernel to use updated packages.

[notice] A new release of pip is available: 25.0.1 -> 25.1.1
[notice] To update, run: python.exe -m pip install --upgrade pip

Requirement already satisfied: sentence_transformers in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (4.0.2)
Requirement already satisfied: transformers<5.0.0,>=4.41.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from sentence_transformers) (4.49.0)
Requirement already satisfied: tqdm in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from sentence_transformers) (4.67.1)
Requirement already satisfied: torch>=1.11.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from sentence_transformers) (2.7.0)
Requirement already satisfied: scikit-learn in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from sentence_transformers) (1.6.1)
Requirement already satisfied: scipy in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from sentence_transformers) (1.15.2)
Requirement already satisfied: huggingface-hub>=0.20.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from sentence_transformers) (0.30.1)
Requirement already satisfied: Pillow in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from sentence_transformers) (11.1.0)
Requirement already satisfied: typing_extensions>=4.5.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from sentence_transformers) (4.12.2)
Requirement already satisfied: filelock in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from huggingface-hub>=0.20.0->sentence_transformers) (3.18.0)
Requirement already satisfied: fsspec>=2023.5.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from huggingface-hub>=0.20.0->sentence_transformers) (2025.3.2)
Requirement already satisfied: packaging>=20.9 in c:\users\konsa\appdata\roaming\python\python313\site-packages (from huggingface-hub>=0.20.0->sentence_transformers) (24.2)
Requirement already satisfied: pyyaml>=5.1 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from huggingface-hub>=0.20.0->sentence_transformers) (6.0.2)
Requirement already satisfied: requests in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from huggingface-hub>=0.20.0->sentence_transformers) (2.32.3)
Requirement already satisfied: sympy>=1.13.3 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from torch>=1.11.0->sentence_transformers) (1.14.0)
Requirement already satisfied: networkx in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from torch>=1.11.0->sentence_transformers) (3.4.2)
Requirement already satisfied: jinja2 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from torch>=1.11.0->sentence_transformers) (3.1.6)
Requirement already satisfied: setuptools in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from torch>=1.11.0->sentence_transformers) (78.1.0)
Requirement already satisfied: colorama in c:\users\konsa\appdata\roaming\python\python313\site-packages (from tqdm->sentence_transformers) (0.4.6)
Requirement already satisfied: numpy>=1.17 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from transformers<5.0.0,>=4.41.0->sentence_transformers) (2.2.4)
Requirement already satisfied: regex!=2019.12.17 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from transformers<5.0.0,>=4.41.0->sentence_transformers) (2024.11.6)
Requirement already satisfied: tokenizers<0.22,>=0.21 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from transformers<5.0.0,>=4.41.0->sentence_transformers) (0.21.1)
Requirement already satisfied: safetensors>=0.4.1 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from transformers<5.0.0,>=4.41.0->sentence_transformers) (0.5.3)
Requirement already satisfied: joblib>=1.2.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from scikit-learn->sentence_transformers) (1.4.2)
Requirement already satisfied: threadpoolctl>=3.1.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from scikit-learn->sentence_transformers) (3.6.0)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from sympy>=1.13.3->torch>=1.11.0->sentence_transformers) (1.3.0)
Requirement already satisfied: MarkupSafe>=2.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from jinja2->torch>=1.11.0->sentence_transformers) (3.0.2)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from requests->huggingface-hub>=0.20.0->sentence_transformers) (3.4.1)
Requirement already satisfied: idna<4,>=2.5 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from requests->huggingface-hub>=0.20.0->sentence_transformers) (3.10)
Requirement already satisfied: urllib3<3,>=1.21.1 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from requests->huggingface-hub>=0.20.0->sentence_transformers) (2.3.0)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from requests->huggingface-hub>=0.20.0->sentence_transformers) (2025.1.31)
Note: you may need to restart the kernel to use updated packages.

[notice] A new release of pip is available: 25.0.1 -> 25.1.1
[notice] To update, run: python.exe -m pip install --upgrade pip

Requirement already satisfied: pip in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (25.0.1)
Collecting pip
  Downloading pip-25.1.1-py3-none-any.whl.metadata (3.6 kB)
Downloading pip-25.1.1-py3-none-any.whl (1.8 MB)
   ---------------------------------------- 0.0/1.8 MB ? eta -:--:--
   ---------------------------------------- 1.8/1.8 MB 17.2 MB/s eta 0:00:00
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 25.0.1
    Uninstalling pip-25.0.1:
      Successfully uninstalled pip-25.0.1
Successfully installed pip-25.1.1
Note: you may need to restart the kernel to use updated packages.
Requirement already satisfied: numpy in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (2.2.4)
Requirement already satisfied: pandas in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (2.2.3)
Requirement already satisfied: scikit-learn in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (1.6.1)
Requirement already satisfied: umap-learn in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (0.5.7)
Requirement already satisfied: cython in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (3.0.12)
Requirement already satisfied: python-dateutil>=2.8.2 in c:\users\konsa\appdata\roaming\python\python313\site-packages (from pandas) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from pandas) (2025.1)
Requirement already satisfied: tzdata>=2022.7 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from pandas) (2025.1)
Requirement already satisfied: scipy>=1.6.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from scikit-learn) (1.15.2)
Requirement already satisfied: joblib>=1.2.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from scikit-learn) (1.4.2)
Requirement already satisfied: threadpoolctl>=3.1.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from scikit-learn) (3.6.0)
Requirement already satisfied: numba>=0.51.2 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from umap-learn) (0.61.2)
Requirement already satisfied: pynndescent>=0.5 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from umap-learn) (0.5.13)
Requirement already satisfied: tqdm in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from umap-learn) (4.67.1)
Requirement already satisfied: llvmlite<0.45,>=0.44.0dev0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from numba>=0.51.2->umap-learn) (0.44.0)
Requirement already satisfied: six>=1.5 in c:\users\konsa\appdata\roaming\python\python313\site-packages (from python-dateutil>=2.8.2->pandas) (1.17.0)
Requirement already satisfied: colorama in c:\users\konsa\appdata\roaming\python\python313\site-packages (from tqdm->umap-learn) (0.4.6)
Note: you may need to restart the kernel to use updated packages.
Requirement already satisfied: hdbscan in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (0.8.40)
Requirement already satisfied: numpy<3,>=1.20 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from hdbscan) (2.2.4)
Requirement already satisfied: scipy>=1.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from hdbscan) (1.15.2)
Requirement already satisfied: scikit-learn>=0.20 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from hdbscan) (1.6.1)
Requirement already satisfied: joblib>=1.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from hdbscan) (1.4.2)
Requirement already satisfied: threadpoolctl>=3.1.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from scikit-learn>=0.20->hdbscan) (3.6.0)
Note: you may need to restart the kernel to use updated packages.
Requirement already satisfied: bertopic in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (0.17.0)
Requirement already satisfied: hdbscan>=0.8.29 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from bertopic) (0.8.40)
Requirement already satisfied: numpy>=1.20.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from bertopic) (2.2.4)
Requirement already satisfied: pandas>=1.1.5 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from bertopic) (2.2.3)
Requirement already satisfied: plotly>=4.7.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from bertopic) (6.0.1)
Requirement already satisfied: scikit-learn>=1.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from bertopic) (1.6.1)
Requirement already satisfied: sentence-transformers>=0.4.1 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from bertopic) (4.0.2)
Requirement already satisfied: tqdm>=4.41.1 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from bertopic) (4.67.1)
Requirement already satisfied: umap-learn>=0.5.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from bertopic) (0.5.7)
Requirement already satisfied: scipy>=1.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from hdbscan>=0.8.29->bertopic) (1.15.2)
Requirement already satisfied: joblib>=1.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from hdbscan>=0.8.29->bertopic) (1.4.2)
Requirement already satisfied: python-dateutil>=2.8.2 in c:\users\konsa\appdata\roaming\python\python313\site-packages (from pandas>=1.1.5->bertopic) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from pandas>=1.1.5->bertopic) (2025.1)
Requirement already satisfied: tzdata>=2022.7 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from pandas>=1.1.5->bertopic) (2025.1)
Requirement already satisfied: narwhals>=1.15.1 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from plotly>=4.7.0->bertopic) (1.34.0)
Requirement already satisfied: packaging in c:\users\konsa\appdata\roaming\python\python313\site-packages (from plotly>=4.7.0->bertopic) (24.2)
Requirement already satisfied: six>=1.5 in c:\users\konsa\appdata\roaming\python\python313\site-packages (from python-dateutil>=2.8.2->pandas>=1.1.5->bertopic) (1.17.0)
Requirement already satisfied: threadpoolctl>=3.1.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from scikit-learn>=1.0->bertopic) (3.6.0)
Requirement already satisfied: transformers<5.0.0,>=4.41.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from sentence-transformers>=0.4.1->bertopic) (4.49.0)
Requirement already satisfied: torch>=1.11.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from sentence-transformers>=0.4.1->bertopic) (2.7.0)
Requirement already satisfied: huggingface-hub>=0.20.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from sentence-transformers>=0.4.1->bertopic) (0.30.1)
Requirement already satisfied: Pillow in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from sentence-transformers>=0.4.1->bertopic) (11.1.0)
Requirement already satisfied: typing_extensions>=4.5.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from sentence-transformers>=0.4.1->bertopic) (4.12.2)
Requirement already satisfied: filelock in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from transformers<5.0.0,>=4.41.0->sentence-transformers>=0.4.1->bertopic) (3.18.0)
Requirement already satisfied: pyyaml>=5.1 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from transformers<5.0.0,>=4.41.0->sentence-transformers>=0.4.1->bertopic) (6.0.2)
Requirement already satisfied: regex!=2019.12.17 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from transformers<5.0.0,>=4.41.0->sentence-transformers>=0.4.1->bertopic) (2024.11.6)
Requirement already satisfied: requests in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from transformers<5.0.0,>=4.41.0->sentence-transformers>=0.4.1->bertopic) (2.32.3)
Requirement already satisfied: tokenizers<0.22,>=0.21 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from transformers<5.0.0,>=4.41.0->sentence-transformers>=0.4.1->bertopic) (0.21.1)
Requirement already satisfied: safetensors>=0.4.1 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from transformers<5.0.0,>=4.41.0->sentence-transformers>=0.4.1->bertopic) (0.5.3)
Requirement already satisfied: fsspec>=2023.5.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from huggingface-hub>=0.20.0->sentence-transformers>=0.4.1->bertopic) (2025.3.2)
Requirement already satisfied: sympy>=1.13.3 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from torch>=1.11.0->sentence-transformers>=0.4.1->bertopic) (1.14.0)
Requirement already satisfied: networkx in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from torch>=1.11.0->sentence-transformers>=0.4.1->bertopic) (3.4.2)
Requirement already satisfied: jinja2 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from torch>=1.11.0->sentence-transformers>=0.4.1->bertopic) (3.1.6)
Requirement already satisfied: setuptools in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from torch>=1.11.0->sentence-transformers>=0.4.1->bertopic) (78.1.0)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from sympy>=1.13.3->torch>=1.11.0->sentence-transformers>=0.4.1->bertopic) (1.3.0)
Requirement already satisfied: colorama in c:\users\konsa\appdata\roaming\python\python313\site-packages (from tqdm>=4.41.1->bertopic) (0.4.6)
Requirement already satisfied: numba>=0.51.2 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from umap-learn>=0.5.0->bertopic) (0.61.2)
Requirement already satisfied: pynndescent>=0.5 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from umap-learn>=0.5.0->bertopic) (0.5.13)
Requirement already satisfied: llvmlite<0.45,>=0.44.0dev0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from numba>=0.51.2->umap-learn>=0.5.0->bertopic) (0.44.0)
Requirement already satisfied: MarkupSafe>=2.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from jinja2->torch>=1.11.0->sentence-transformers>=0.4.1->bertopic) (3.0.2)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from requests->transformers<5.0.0,>=4.41.0->sentence-transformers>=0.4.1->bertopic) (3.4.1)
Requirement already satisfied: idna<4,>=2.5 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from requests->transformers<5.0.0,>=4.41.0->sentence-transformers>=0.4.1->bertopic) (3.10)
Requirement already satisfied: urllib3<3,>=1.21.1 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from requests->transformers<5.0.0,>=4.41.0->sentence-transformers>=0.4.1->bertopic) (2.3.0)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from requests->transformers<5.0.0,>=4.41.0->sentence-transformers>=0.4.1->bertopic) (2025.1.31)
Note: you may need to restart the kernel to use updated packages.
Requirement already satisfied: vaderSentiment in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (3.3.2)
Requirement already satisfied: requests in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from vaderSentiment) (2.32.3)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from requests->vaderSentiment) (3.4.1)
Requirement already satisfied: idna<4,>=2.5 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from requests->vaderSentiment) (3.10)
Requirement already satisfied: urllib3<3,>=1.21.1 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from requests->vaderSentiment) (2.3.0)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from requests->vaderSentiment) (2025.1.31)
Note: you may need to restart the kernel to use updated packages.
Requirement already satisfied: accelerate in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (1.6.0)
Requirement already satisfied: numpy<3.0.0,>=1.17 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from accelerate) (2.2.4)
Requirement already satisfied: packaging>=20.0 in c:\users\konsa\appdata\roaming\python\python313\site-packages (from accelerate) (24.2)
Requirement already satisfied: psutil in c:\users\konsa\appdata\roaming\python\python313\site-packages (from accelerate) (7.0.0)
Requirement already satisfied: pyyaml in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from accelerate) (6.0.2)
Requirement already satisfied: torch>=2.0.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from accelerate) (2.7.0)
Requirement already satisfied: huggingface-hub>=0.21.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from accelerate) (0.30.1)
Requirement already satisfied: safetensors>=0.4.3 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from accelerate) (0.5.3)
Requirement already satisfied: filelock in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from huggingface-hub>=0.21.0->accelerate) (3.18.0)
Requirement already satisfied: fsspec>=2023.5.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from huggingface-hub>=0.21.0->accelerate) (2025.3.2)
Requirement already satisfied: requests in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from huggingface-hub>=0.21.0->accelerate) (2.32.3)
Requirement already satisfied: tqdm>=4.42.1 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from huggingface-hub>=0.21.0->accelerate) (4.67.1)
Requirement already satisfied: typing-extensions>=3.7.4.3 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from huggingface-hub>=0.21.0->accelerate) (4.12.2)
Requirement already satisfied: sympy>=1.13.3 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from torch>=2.0.0->accelerate) (1.14.0)
Requirement already satisfied: networkx in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from torch>=2.0.0->accelerate) (3.4.2)
Requirement already satisfied: jinja2 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from torch>=2.0.0->accelerate) (3.1.6)
Requirement already satisfied: setuptools in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from torch>=2.0.0->accelerate) (78.1.0)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from sympy>=1.13.3->torch>=2.0.0->accelerate) (1.3.0)
Requirement already satisfied: colorama in c:\users\konsa\appdata\roaming\python\python313\site-packages (from tqdm>=4.42.1->huggingface-hub>=0.21.0->accelerate) (0.4.6)
Requirement already satisfied: MarkupSafe>=2.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from jinja2->torch>=2.0.0->accelerate) (3.0.2)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from requests->huggingface-hub>=0.21.0->accelerate) (3.4.1)
Requirement already satisfied: idna<4,>=2.5 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from requests->huggingface-hub>=0.21.0->accelerate) (3.10)
Requirement already satisfied: urllib3<3,>=1.21.1 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from requests->huggingface-hub>=0.21.0->accelerate) (2.3.0)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from requests->huggingface-hub>=0.21.0->accelerate) (2025.1.31)
Note: you may need to restart the kernel to use updated packages.
Requirement already satisfied: transformers==4.49.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (4.49.0)
Requirement already satisfied: filelock in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from transformers==4.49.0) (3.18.0)
Requirement already satisfied: huggingface-hub<1.0,>=0.26.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from transformers==4.49.0) (0.30.1)
Requirement already satisfied: numpy>=1.17 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from transformers==4.49.0) (2.2.4)
Requirement already satisfied: packaging>=20.0 in c:\users\konsa\appdata\roaming\python\python313\site-packages (from transformers==4.49.0) (24.2)
Requirement already satisfied: pyyaml>=5.1 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from transformers==4.49.0) (6.0.2)
Requirement already satisfied: regex!=2019.12.17 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from transformers==4.49.0) (2024.11.6)
Requirement already satisfied: requests in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from transformers==4.49.0) (2.32.3)
Requirement already satisfied: tokenizers<0.22,>=0.21 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from transformers==4.49.0) (0.21.1)
Requirement already satisfied: safetensors>=0.4.1 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from transformers==4.49.0) (0.5.3)
Requirement already satisfied: tqdm>=4.27 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from transformers==4.49.0) (4.67.1)
Requirement already satisfied: fsspec>=2023.5.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from huggingface-hub<1.0,>=0.26.0->transformers==4.49.0) (2025.3.2)
Requirement already satisfied: typing-extensions>=3.7.4.3 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from huggingface-hub<1.0,>=0.26.0->transformers==4.49.0) (4.12.2)
Requirement already satisfied: colorama in c:\users\konsa\appdata\roaming\python\python313\site-packages (from tqdm>=4.27->transformers==4.49.0) (0.4.6)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from requests->transformers==4.49.0) (3.4.1)
Requirement already satisfied: idna<4,>=2.5 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from requests->transformers==4.49.0) (3.10)
Requirement already satisfied: urllib3<3,>=1.21.1 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from requests->transformers==4.49.0) (2.3.0)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from requests->transformers==4.49.0) (2025.1.31)
Note: you may need to restart the kernel to use updated packages.
Requirement already satisfied: keybert in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (0.9.0)
Requirement already satisfied: sentence-transformers in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (4.0.2)
Requirement already satisfied: numpy>=1.18.5 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from keybert) (2.2.4)
Requirement already satisfied: rich>=10.4.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from keybert) (14.0.0)
Requirement already satisfied: scikit-learn>=0.22.2 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from keybert) (1.6.1)
Requirement already satisfied: transformers<5.0.0,>=4.41.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from sentence-transformers) (4.49.0)
Requirement already satisfied: tqdm in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from sentence-transformers) (4.67.1)
Requirement already satisfied: torch>=1.11.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from sentence-transformers) (2.7.0)
Requirement already satisfied: scipy in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from sentence-transformers) (1.15.2)
Requirement already satisfied: huggingface-hub>=0.20.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from sentence-transformers) (0.30.1)
Requirement already satisfied: Pillow in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from sentence-transformers) (11.1.0)
Requirement already satisfied: typing_extensions>=4.5.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from sentence-transformers) (4.12.2)
Requirement already satisfied: filelock in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from transformers<5.0.0,>=4.41.0->sentence-transformers) (3.18.0)
Requirement already satisfied: packaging>=20.0 in c:\users\konsa\appdata\roaming\python\python313\site-packages (from transformers<5.0.0,>=4.41.0->sentence-transformers) (24.2)
Requirement already satisfied: pyyaml>=5.1 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from transformers<5.0.0,>=4.41.0->sentence-transformers) (6.0.2)
Requirement already satisfied: regex!=2019.12.17 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from transformers<5.0.0,>=4.41.0->sentence-transformers) (2024.11.6)
Requirement already satisfied: requests in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from transformers<5.0.0,>=4.41.0->sentence-transformers) (2.32.3)
Requirement already satisfied: tokenizers<0.22,>=0.21 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from transformers<5.0.0,>=4.41.0->sentence-transformers) (0.21.1)
Requirement already satisfied: safetensors>=0.4.1 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from transformers<5.0.0,>=4.41.0->sentence-transformers) (0.5.3)
Requirement already satisfied: fsspec>=2023.5.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from huggingface-hub>=0.20.0->sentence-transformers) (2025.3.2)
Requirement already satisfied: markdown-it-py>=2.2.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from rich>=10.4.0->keybert) (3.0.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in c:\users\konsa\appdata\roaming\python\python313\site-packages (from rich>=10.4.0->keybert) (2.19.1)
Requirement already satisfied: mdurl~=0.1 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from markdown-it-py>=2.2.0->rich>=10.4.0->keybert) (0.1.2)
Requirement already satisfied: joblib>=1.2.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from scikit-learn>=0.22.2->keybert) (1.4.2)
Requirement already satisfied: threadpoolctl>=3.1.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from scikit-learn>=0.22.2->keybert) (3.6.0)
Requirement already satisfied: sympy>=1.13.3 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from torch>=1.11.0->sentence-transformers) (1.14.0)
Requirement already satisfied: networkx in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from torch>=1.11.0->sentence-transformers) (3.4.2)
Requirement already satisfied: jinja2 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from torch>=1.11.0->sentence-transformers) (3.1.6)
Requirement already satisfied: setuptools in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from torch>=1.11.0->sentence-transformers) (78.1.0)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from sympy>=1.13.3->torch>=1.11.0->sentence-transformers) (1.3.0)
Requirement already satisfied: colorama in c:\users\konsa\appdata\roaming\python\python313\site-packages (from tqdm->sentence-transformers) (0.4.6)
Requirement already satisfied: MarkupSafe>=2.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from jinja2->torch>=1.11.0->sentence-transformers) (3.0.2)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from requests->transformers<5.0.0,>=4.41.0->sentence-transformers) (3.4.1)
Requirement already satisfied: idna<4,>=2.5 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from requests->transformers<5.0.0,>=4.41.0->sentence-transformers) (3.10)
Requirement already satisfied: urllib3<3,>=1.21.1 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from requests->transformers<5.0.0,>=4.41.0->sentence-transformers) (2.3.0)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from requests->transformers<5.0.0,>=4.41.0->sentence-transformers) (2025.1.31)
Note: you may need to restart the kernel to use updated packages.
Requirement already satisfied: bertopic[all] in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (0.17.0)
Requirement already satisfied: hdbscan>=0.8.29 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from bertopic[all]) (0.8.40)
Requirement already satisfied: numpy>=1.20.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from bertopic[all]) (2.2.4)
Requirement already satisfied: pandas>=1.1.5 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from bertopic[all]) (2.2.3)
Requirement already satisfied: plotly>=4.7.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from bertopic[all]) (6.0.1)
Requirement already satisfied: scikit-learn>=1.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from bertopic[all]) (1.6.1)
Requirement already satisfied: sentence-transformers>=0.4.1 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from bertopic[all]) (4.0.2)
Requirement already satisfied: tqdm>=4.41.1 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from bertopic[all]) (4.67.1)
Requirement already satisfied: umap-learn>=0.5.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from bertopic[all]) (0.5.7)
Requirement already satisfied: scipy>=1.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from hdbscan>=0.8.29->bertopic[all]) (1.15.2)
Requirement already satisfied: joblib>=1.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from hdbscan>=0.8.29->bertopic[all]) (1.4.2)
Requirement already satisfied: python-dateutil>=2.8.2 in c:\users\konsa\appdata\roaming\python\python313\site-packages (from pandas>=1.1.5->bertopic[all]) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from pandas>=1.1.5->bertopic[all]) (2025.1)
Requirement already satisfied: tzdata>=2022.7 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from pandas>=1.1.5->bertopic[all]) (2025.1)
Requirement already satisfied: narwhals>=1.15.1 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from plotly>=4.7.0->bertopic[all]) (1.34.0)
Requirement already satisfied: packaging in c:\users\konsa\appdata\roaming\python\python313\site-packages (from plotly>=4.7.0->bertopic[all]) (24.2)
Requirement already satisfied: six>=1.5 in c:\users\konsa\appdata\roaming\python\python313\site-packages (from python-dateutil>=2.8.2->pandas>=1.1.5->bertopic[all]) (1.17.0)
Requirement already satisfied: threadpoolctl>=3.1.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from scikit-learn>=1.0->bertopic[all]) (3.6.0)
Requirement already satisfied: transformers<5.0.0,>=4.41.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from sentence-transformers>=0.4.1->bertopic[all]) (4.49.0)
Requirement already satisfied: torch>=1.11.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from sentence-transformers>=0.4.1->bertopic[all]) (2.7.0)
Requirement already satisfied: huggingface-hub>=0.20.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from sentence-transformers>=0.4.1->bertopic[all]) (0.30.1)
Requirement already satisfied: Pillow in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from sentence-transformers>=0.4.1->bertopic[all]) (11.1.0)
Requirement already satisfied: typing_extensions>=4.5.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from sentence-transformers>=0.4.1->bertopic[all]) (4.12.2)
Requirement already satisfied: filelock in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from transformers<5.0.0,>=4.41.0->sentence-transformers>=0.4.1->bertopic[all]) (3.18.0)
Requirement already satisfied: pyyaml>=5.1 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from transformers<5.0.0,>=4.41.0->sentence-transformers>=0.4.1->bertopic[all]) (6.0.2)
Requirement already satisfied: regex!=2019.12.17 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from transformers<5.0.0,>=4.41.0->sentence-transformers>=0.4.1->bertopic[all]) (2024.11.6)
Requirement already satisfied: requests in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from transformers<5.0.0,>=4.41.0->sentence-transformers>=0.4.1->bertopic[all]) (2.32.3)
Requirement already satisfied: tokenizers<0.22,>=0.21 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from transformers<5.0.0,>=4.41.0->sentence-transformers>=0.4.1->bertopic[all]) (0.21.1)
Requirement already satisfied: safetensors>=0.4.1 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from transformers<5.0.0,>=4.41.0->sentence-transformers>=0.4.1->bertopic[all]) (0.5.3)
Requirement already satisfied: fsspec>=2023.5.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from huggingface-hub>=0.20.0->sentence-transformers>=0.4.1->bertopic[all]) (2025.3.2)
Requirement already satisfied: sympy>=1.13.3 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from torch>=1.11.0->sentence-transformers>=0.4.1->bertopic[all]) (1.14.0)
Requirement already satisfied: networkx in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from torch>=1.11.0->sentence-transformers>=0.4.1->bertopic[all]) (3.4.2)
Requirement already satisfied: jinja2 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from torch>=1.11.0->sentence-transformers>=0.4.1->bertopic[all]) (3.1.6)
Requirement already satisfied: setuptools in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from torch>=1.11.0->sentence-transformers>=0.4.1->bertopic[all]) (78.1.0)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from sympy>=1.13.3->torch>=1.11.0->sentence-transformers>=0.4.1->bertopic[all]) (1.3.0)
Requirement already satisfied: colorama in c:\users\konsa\appdata\roaming\python\python313\site-packages (from tqdm>=4.41.1->bertopic[all]) (0.4.6)
Requirement already satisfied: numba>=0.51.2 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from umap-learn>=0.5.0->bertopic[all]) (0.61.2)
Requirement already satisfied: pynndescent>=0.5 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from umap-learn>=0.5.0->bertopic[all]) (0.5.13)
Requirement already satisfied: llvmlite<0.45,>=0.44.0dev0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from numba>=0.51.2->umap-learn>=0.5.0->bertopic[all]) (0.44.0)
Requirement already satisfied: MarkupSafe>=2.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from jinja2->torch>=1.11.0->sentence-transformers>=0.4.1->bertopic[all]) (3.0.2)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from requests->transformers<5.0.0,>=4.41.0->sentence-transformers>=0.4.1->bertopic[all]) (3.4.1)
Requirement already satisfied: idna<4,>=2.5 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from requests->transformers<5.0.0,>=4.41.0->sentence-transformers>=0.4.1->bertopic[all]) (3.10)
Requirement already satisfied: urllib3<3,>=1.21.1 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from requests->transformers<5.0.0,>=4.41.0->sentence-transformers>=0.4.1->bertopic[all]) (2.3.0)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from requests->transformers<5.0.0,>=4.41.0->sentence-transformers>=0.4.1->bertopic[all]) (2025.1.31)
Note: you may need to restart the kernel to use updated packages.

WARNING: bertopic 0.17.0 does not provide the extra 'all'

Requirement already satisfied: wheel in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (0.45.1)
Note: you may need to restart the kernel to use updated packages.
Requirement already satisfied: cython in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (3.0.12)
Note: you may need to restart the kernel to use updated packages.
Requirement already satisfied: numpy in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (2.2.4)
Note: you may need to restart the kernel to use updated packages.
Requirement already satisfied: hdbscan in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (0.8.40)
Requirement already satisfied: numpy<3,>=1.20 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from hdbscan) (2.2.4)
Requirement already satisfied: scipy>=1.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from hdbscan) (1.15.2)
Requirement already satisfied: scikit-learn>=0.20 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from hdbscan) (1.6.1)
Requirement already satisfied: joblib>=1.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from hdbscan) (1.4.2)
Requirement already satisfied: threadpoolctl>=3.1.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from scikit-learn>=0.20->hdbscan) (3.6.0)
Note: you may need to restart the kernel to use updated packages.
Requirement already satisfied: nltk in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (3.9.1)
Requirement already satisfied: click in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from nltk) (8.1.8)
Requirement already satisfied: joblib in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from nltk) (1.4.2)
Requirement already satisfied: regex>=2021.8.3 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from nltk) (2024.11.6)
Requirement already satisfied: tqdm in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from nltk) (4.67.1)
Requirement already satisfied: colorama in c:\users\konsa\appdata\roaming\python\python313\site-packages (from click->nltk) (0.4.6)
Note: you may need to restart the kernel to use updated packages.
Requirement already satisfied: stanza in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (1.10.1)
Requirement already satisfied: emoji in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from stanza) (2.14.1)
Requirement already satisfied: numpy in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from stanza) (2.2.4)
Requirement already satisfied: protobuf>=3.15.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from stanza) (6.30.2)
Requirement already satisfied: requests in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from stanza) (2.32.3)
Requirement already satisfied: networkx in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from stanza) (3.4.2)
Requirement already satisfied: torch>=1.3.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from stanza) (2.7.0)
Requirement already satisfied: tqdm in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from stanza) (4.67.1)
Requirement already satisfied: filelock in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from torch>=1.3.0->stanza) (3.18.0)
Requirement already satisfied: typing-extensions>=4.10.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from torch>=1.3.0->stanza) (4.12.2)
Requirement already satisfied: sympy>=1.13.3 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from torch>=1.3.0->stanza) (1.14.0)
Requirement already satisfied: jinja2 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from torch>=1.3.0->stanza) (3.1.6)
Requirement already satisfied: fsspec in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from torch>=1.3.0->stanza) (2025.3.2)
Requirement already satisfied: setuptools in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from torch>=1.3.0->stanza) (78.1.0)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from sympy>=1.13.3->torch>=1.3.0->stanza) (1.3.0)
Requirement already satisfied: MarkupSafe>=2.0 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from jinja2->torch>=1.3.0->stanza) (3.0.2)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from requests->stanza) (3.4.1)
Requirement already satisfied: idna<4,>=2.5 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from requests->stanza) (3.10)
Requirement already satisfied: urllib3<3,>=1.21.1 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from requests->stanza) (2.3.0)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\konsa\appdata\local\programs\python\python313\lib\site-packages (from requests->stanza) (2025.1.31)
Requirement already satisfied: colorama in c:\users\konsa\appdata\roaming\python\python313\site-packages (from tqdm->stanza) (0.4.6)
Note: you may need to restart the kernel to use updated packages.

import pandas as pd

file = "reviews_df.csv"
reviews_df = pd.read_csv(file, encoding='utf-8')
reviews_df.head()

print("Total number of reviews:", len(reviews_df))

Total number of reviews: 656850

import torch

torch.cuda.set_device(0)  # Ensure correct GPU

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[7], line 3
      1 import torch
----> 3 torch.cuda.set_device(0)  # Ensure correct GPU

File c:\Users\konsa\AppData\Local\Programs\Python\Python313\Lib\site-packages\torch\cuda\__init__.py:529, in set_device(device)
    527 device = _get_device_index(device)
    528 if device >= 0:
--> 529     torch._C._cuda_setDevice(device)

AttributeError: module 'torch._C' has no attribute '_cuda_setDevice'

import re
#Data cleaning
comment_col='Comment'
# Drop empty or NaN comments
reviews_df = reviews_df.dropna(subset=[comment_col])
reviews_df[comment_col] = reviews_df[comment_col].astype(str)

# Remove whitespace-only comments
reviews_df = reviews_df[reviews_df[comment_col].str.strip() != ""]

# Remove short comments (you can tweak the length as needed)
reviews_df = reviews_df[reviews_df[comment_col].str.len() > 10]

# Remove comments that are mostly symbols or gibberish
def is_gibberish(text):
    # Remove spaces and check if remaining text is mostly non-alphanumeric
    cleaned = re.sub(r'\s+', '', text)
    ratio = len(re.sub(r'\w', '', cleaned)) / len(cleaned) if len(cleaned) > 0 else 1
    return ratio > 0.5

reviews_df = reviews_df[~reviews_df[comment_col].apply(is_gibberish)]

print(len(reviews_df))

651426

# Make sure the review_date column is in datetime format
reviews_df["Date"] = pd.to_datetime(reviews_df["Date"], errors='coerce')

# Filter out rows where the date is in 2015 or earlier
reviews_df = reviews_df[reviews_df["Date"] > "2015-12-31"]

print(len(reviews_df))

244936

# Ensure numeric types (in case they're read as strings)
reviews_df["Thumbs Up"] = pd.to_numeric(reviews_df["Thumbs Up"], errors="coerce").fillna(0)
reviews_df["Thumbs Down"] = pd.to_numeric(reviews_df["Thumbs Down"], errors="coerce").fillna(0)

# Avoid division by zero by replacing 0 with NaN temporarily
reviews_df["total_votes"] = reviews_df["Thumbs Up"] + reviews_df["Thumbs Down"]
reviews_df["thumbs_down_ratio"] = reviews_df["Thumbs Down"] / reviews_df["total_votes"]
reviews_df["thumbs_down_ratio"] = reviews_df["thumbs_down_ratio"].fillna(0)  # Optionally fill back with 0

# Apply condition safely
condition = (
    (reviews_df["Thumbs Down"] > 3) &
    (reviews_df["thumbs_down_ratio"] > 0.5)
)

# Remove reviews that meet both conditions
reviews_df = reviews_df[~condition]

print(len(reviews_df))

237371

reviews_df.reset_index(drop=True, inplace=True)

reviews_df.head()

from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer

# Initialize sentiment analyzer
analyzer = SentimentIntensityAnalyzer()

# Apply sentiment analysis
reviews_df['sentiment_score'] = reviews_df['Comment'].apply(lambda x: analyzer.polarity_scores(x)['compound'])

# Optional: classify as positive/neutral/negative
reviews_df['sentiment_label'] = reviews_df['sentiment_score'].apply(
    lambda x: 'positive' if x > 0.05 else 'negative' if x < -0.05 else 'neutral'
)

import torch
from transformers import pipeline

# Load the sentiment analysis pipeline with the correct model
classifier = pipeline("sentiment-analysis", model="distilbert/distilbert-base-uncased-finetuned-sst-2-english",device=0)

# Convert comments to list
comments = list(reviews_df['Comment'])

# Run batched sentiment analysis
results = classifier(comments, batch_size=8)

# Extract and assign results
reviews_df['sentiment_label_transformer'] = [r['label'] for r in results]
reviews_df['sentiment_score_transformer'] = [r['score'] for r in results]

2025-05-09 18:04:31.664844: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1746813871.851135      31 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1746813871.905001      31 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered

config.json:   0%|          | 0.00/629 [00:00<?, ?B/s]

Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`

model.safetensors:   0%|          | 0.00/268M [00:00<?, ?B/s]

tokenizer_config.json:   0%|          | 0.00/48.0 [00:00<?, ?B/s]

vocab.txt:   0%|          | 0.00/232k [00:00<?, ?B/s]

Device set to use cuda:0

reviews_df.head()

#Sentiment models evaluation

print(reviews_df[reviews_df['Quality'] == 1].groupby('sentiment_label').size())
print(reviews_df[reviews_df['Quality'] == 1].groupby('sentiment_label_transformer').size())

reviews_df = reviews_df.drop(['sentiment_score', 'sentiment_label','total_votes','thumbs_down_ratio'], axis=1)

print(reviews_df[reviews_df['Quality'] == 5].groupby('sentiment_label_transformer').size())

final_df=reviews_df
final_df.to_csv("final_df.csv", index=False)

sentiment_label
negative    31790
neutral      2955
positive    10832
dtype: int64
sentiment_label_transformer
NEGATIVE    42813
POSITIVE     2764
dtype: int64
sentiment_label_transformer
NEGATIVE      7177
POSITIVE    120637
dtype: int64

final_df

import nltk
from nltk.corpus import stopwords
from collections import Counter
import re

nltk.download('stopwords')

# Basic stopwords
stop_words = set(stopwords.words('english'))

# Add custom stopwords for reviews
custom_stopwords = ['class', 'professor', 'he', 'she', 'his', 'her', 'very', 'really', 'one', 'get', 'prof',
                    'got', 'take', 'took', 'like','also','2','3','baruch']
stop_words.update(custom_stopwords)

# Combine all comments into 1 string
all_text = " ".join(final_df['Comment'].fillna("").astype(str).tolist())

# Tokenize & Clean
words = re.findall(r'\b\w+\b', all_text.lower())

# Remove stopwords
filtered_words = [word for word in words if word not in stop_words]

# Count frequency
word_freq = Counter(filtered_words)

# Top 30 words
print(word_freq.most_common(20))

[nltk_data] Downloading package stopwords to /usr/share/nltk_data...
[nltk_data]   Package stopwords is already up-to-date!

[('students', 49542), ('easy', 47793), ('work', 44473), ('great', 40222), ('good', 40200), ('gives', 37016), ('best', 36440), ('lot', 34269), ('lectures', 33924), ('exams', 31604), ('time', 31478), ('would', 31307), ('final', 29157), ('grade', 27807), ('hard', 25441), ('questions', 25313), ('assignments', 24404), ('well', 24061), ('help', 23953), ('makes', 23461)]

from sentence_transformers import SentenceTransformer
import pandas as pd
model = SentenceTransformer('all-MiniLM-L6-v2')  # or any SBERT model

sentences = final_df['Comment'].fillna("").astype(str).tolist()
embeddings = model.encode(sentences, show_progress_bar=True)

from sklearn.cluster import KMeans

n_clusters = 7
kmeans = KMeans(n_clusters=n_clusters, random_state=42)
labels = kmeans.fit_predict(embeddings)

Batches:   0%|          | 0/7418 [00:00<?, ?it/s]

print(labels)

from keybert import KeyBERT

kw_model = KeyBERT('all-MiniLM-L6-v2')

for cluster_id in set(labels):
    cluster_texts = [text for text, label in zip(sentences, labels) if label == cluster_id]
    joined_text = " ".join(cluster_texts)

    keywords = kw_model.extract_keywords(joined_text, keyphrase_ngram_range=(1,3), top_n=5)
    
    print(f"Cluster {cluster_id}: {[kw[0] for kw in keywords]}")

[5 3 5 ... 6 1 6]
Cluster 0: ['class professor merkin', 'struggles classes professor', 'professor classes extremely', 'classes amazing professor', 'classes wonderful professor']
Cluster 1: ['lectures decent grading', 'grader lectures decent', 'lecture good quizzes', 'lectures quizzes decent', 'lecture grades fairly']
Cluster 2: ['mean hates shes', 'shes mean hates', 'awful lovely professor', 'lady ruin life', 'utterly dreadful woman']
Cluster 3: ['class online difficult', 'online class difficult', 'class online overwhelmed', 'online difficult class', 'online classes hard']
Cluster 4: ['struggling class professor', 'exams challenging professor', 'challenging class professor', 'professor going struggle', 'professor grades terribly']
Cluster 5: ['class difficult shes', 'class difficult teacher', 'class student struggled', 'class teacher difficulty', 'class struggling teacher']
Cluster 6: ['recommend professor rajkumar', 'enjoying lecture professor', 'lectures understanding professor', 'lectures insightful professor', 'lectures fantastic professor']

#Runtime approx 6 hours with P100 accelerator
import stanza
import torch
import pandas as pd
from tqdm import tqdm

# Load your data
df = pd.read_csv("final_df.csv")  # Replace with your actual path
df['objects_only'] = ""  # Optional: create a column for output

# Download models (only once)
stanza.download('en')

# Initialize pipeline with GPU support
torch.cuda.set_device(0)  # Ensure correct GPU
nlp = stanza.Pipeline(lang='en', processors='tokenize,mwt,pos,lemma,depparse', use_gpu=True)

# Process each comment
results = []
for idx, text in tqdm(zip(df.index, df['Comment']), total=len(df)):
    try:
        doc = nlp(text)
        objects = [word.text for sent in doc.sentences for word in sent.words if word.deprel == 'obj']
        results.append(" ".join(objects))
    except Exception:
        results.append("")

df['objects_only'] = results

df.to_csv("student_reviews_with_objects.csv", index=False)

import pandas as pd
final_df = pd.read_csv("/kaggle/input/reviews-w-objects/student_reviews_with_objects.csv")

import nltk
from nltk.corpus import stopwords
from collections import Counter
import re

nltk.download('stopwords')

# Basic stopwords
stop_words = set(stopwords.words('english'))

# Add custom stopwords for reviews
custom_stopwords = ['class', 'professor', 'he', 'she', 'his', 'her', 'very', 'really', 'one', 'get', 'prof',
                    'got', 'take', 'took', 'like','also','2','3','baruch']
stop_words.update(custom_stopwords)

# Combine all comments into 1 string
all_text = " ".join(final_df['objects_only'].fillna("").astype(str).tolist())

# Tokenize & Clean
words = re.findall(r'\b\w+\b', all_text.lower())

# Remove stopwords
filtered_words = [word for word in words if word not in stop_words]

# Count frequency
word_freq = Counter(filtered_words)

# Top 30 words
print(word_freq.most_common(50))

[nltk_data] Downloading package stopwords to /usr/share/nltk_data...
[nltk_data]   Package stopwords is already up-to-date!

[('lot', 18640), ('work', 16357), ('questions', 14282), ('students', 13366), ('grade', 13054), ('material', 11448), ('time', 11120), ('everything', 8817), ('attention', 8029), ('notes', 7866), ('feedback', 7778), ('assignments', 7546), ('course', 6736), ('homework', 6681), ('lectures', 6571), ('credit', 6198), ('us', 6181), ('things', 5915), ('classes', 5291), ('textbook', 5029), ('anything', 4953), ('best', 4265), ('exams', 4187), ('points', 4025), ('quizzes', 3755), ('grades', 3586), ('slides', 3525), ('something', 3520), ('effort', 3518), ('question', 3389), ('book', 3287), ('papers', 3156), ('chance', 2780), ('stuff', 2762), ('readings', 2701), ('help', 2665), ('math', 2652), ('everyone', 2637), ('problems', 2489), ('way', 2396), ('much', 2392), ('concepts', 2390), ('reviews', 2389), ('tests', 2349), ('paper', 2277), ('nothing', 2263), ('topics', 2203), ('sense', 2127), ('lots', 2116), ('lecture', 2048)]

from sentence_transformers import SentenceTransformer
import pandas as pd
model = SentenceTransformer('all-MiniLM-L6-v2')  # or any SBERT model

sentences = final_df['objects_only'].fillna("").astype(str).tolist()
embeddings = model.encode(sentences, show_progress_bar=True)

from sklearn.cluster import KMeans

n_clusters = 7
kmeans = KMeans(n_clusters=n_clusters, random_state=42)
labels = kmeans.fit_predict(embeddings)

Batches:   0%|          | 0/7419 [00:00<?, ?it/s]

print(labels)

from keybert import KeyBERT

kw_model = KeyBERT('all-MiniLM-L6-v2')

for cluster_id in set(labels):
    cluster_texts = [text for text, label in zip(sentences, labels) if label == cluster_id]
    joined_text = " ".join(cluster_texts)

    keywords = kw_model.extract_keywords(joined_text, keyphrase_ngram_range=(1,3), top_n=5)
    
    print(f"Cluster {cluster_id}: {[kw[0] for kw in keywords]}")

[0 3 2 ... 5 5 6]
Cluster 0: ['class homework professor', 'homework classes shes', 'class shes quizzes', 'professor classwork', 'material shes homework']
Cluster 1: ['em']
Cluster 2: ['lecture homework students', 'lecture homeworks exams', 'lecture homework quizzes', 'lecture homework tests', 'lecture people homework']
Cluster 3: ['paper quizzes attention', 'quiz tests attention', 'attention work quizzes', 'quizzes test attention', 'paper work quizzes']
Cluster 4: ['points lots classes', 'points lots class', 'class points lots', 'point class thoughts', 'lot class discussions']
Cluster 5: ['class notes hes', 'guy notes homework', 'homework class professor', 'notes quizzes professor', 'students class hes']
Cluster 6: ['subject words lot', 'lot homework care', 'word work lot', 'words lot material', 'effort brain lots']

device = 'cuda' if torch.cuda.is_available() else 'cpu'
print(device)

cuda

import pandas as pd
import numpy as np
import torch
from sentence_transformers import SentenceTransformer, util
from tqdm import tqdm

labels = ['work','questions','grade','material','time','attention','notes','feedback','assignments','homework','course','lectures','credit',
 'classes','textbook','exams','quizzes','grades','slides','effort','book','papers','readings','help','tests','paper','topics','lecture',
 'hw','fun','test','essays','gpa','exam','emails','assignment','information','examples','videos','subject','quiz','knowledge','instructions',
 'answers','syllabus','attendance','chapters','opportunity','materials','accent','reading','final','opportunities','skills','attitude','midterm',
 'essay','understanding','books','project','participation','projects','content','lessons','labs','advice','lab','mistakes','credits','teaching','style',
 'discussions','email','criteria','passion','environment','powerpoints','homeworks','info','guide','score','expectations','money','articles','respect',
 'presentation','resources','blackboard','phone','directions','curve','opinion','discussion','deadlines','sessions','powerpoint','group','presentations',
 'guidelines','stress','grader','home','tools','response','format','textbooks','schedule','session','note','research','formulas','background',
 'explanations','difficulty','patience','method','terms','terminology','formulas','career','midterms','posts','feedbacks','methods','hws','attempts',
 'ability','scores','structure','explanation','humor','requirements','practice','policy','tutoring','instruction','exercises','classroom','guidance',
 'curriculum','deadline','recordings','absences','pressure','finals','results','clarification']

# Load model and put it on GPU
device = 'cuda' if torch.cuda.is_available() else 'cpu'
model = SentenceTransformer('all-MiniLM-L6-v2', device=device)

# Encode labels once (on GPU)
label_embeddings = model.encode(labels, convert_to_tensor=True, device=device)

# Batch size (adjust depending on your GPU memory)
batch_size = 32

# Prepare list to collect results
similarities_list = []

# Process comments in batches
for i in tqdm(range(0, len(final_df), batch_size), desc="Processing Batches"):
    batch_comments = final_df['objects_only'].iloc[i:i+batch_size].tolist()

    # Encode batch (on GPU)
    comment_embeddings = model.encode(batch_comments, convert_to_tensor=True, device=device)

    # Compute cosine similarities (on GPU)
    cosine_scores = util.cos_sim(comment_embeddings, label_embeddings)

    # Move to CPU and convert to numpy
    batch_similarities = cosine_scores.cpu().numpy()

    similarities_list.append(batch_similarities)

# Stack all batches
all_similarities = np.vstack(similarities_list)

# Create DataFrame
similarities_df = pd.DataFrame(all_similarities, columns=[f"{label}_prob" for label in labels])

# Final merged DataFrame
final_df = pd.concat([final_df.reset_index(drop=True), similarities_df], axis=1)

# Done!
#final_df.to_csv('df_with_labels.csv', index=False)

Batches:   0%|          | 0/5 [00:00<?, ?it/s]

Processing Batches:   0%|          | 0/7419 [00:00<?, ?it/s]

Batches:   0%|          | 0/1 [00:00<?, ?it/s]

Batches:   0%|          | 0/1 [00:00<?, ?it/s]

Batches:   0%|          | 0/1 [00:00<?, ?it/s]

Batches:   0%|          | 0/1 [00:00<?, ?it/s]

Batches:   0%|          | 0/1 [00:00<?, ?it/s]

Processing Batches:   0%|          | 5/7419 [00:00<02:29, 49.55it/s]

Batches:   0%|          | 0/1 [00:00<?, ?it/s]

Batches:   0%|          | 0/1 [00:00<?, ?it/s]

print(list(final_df.columns))

['Unnamed: 0', 'Department', 'Quality', 'Difficulty', 'Comment', 'Date', 'College', 'sentiment_label_transformer', 'sentiment_score_transformer', 'objects_only', 'work_prob', 'questions_prob', 'grade_prob', 'material_prob', 'time_prob', 'attention_prob', 'notes_prob', 'feedback_prob', 'assignments_prob', 'homework_prob', 'course_prob', 'lectures_prob', 'credit_prob', 'classes_prob', 'textbook_prob', 'exams_prob', 'quizzes_prob', 'grades_prob', 'slides_prob', 'effort_prob', 'book_prob', 'papers_prob', 'readings_prob', 'help_prob', 'tests_prob', 'paper_prob', 'topics_prob', 'lecture_prob', 'hw_prob', 'fun_prob', 'test_prob', 'essays_prob', 'gpa_prob', 'exam_prob', 'emails_prob', 'assignment_prob', 'information_prob', 'examples_prob', 'videos_prob', 'subject_prob', 'quiz_prob', 'knowledge_prob', 'instructions_prob', 'answers_prob', 'syllabus_prob', 'attendance_prob', 'chapters_prob', 'opportunity_prob', 'materials_prob', 'accent_prob', 'reading_prob', 'final_prob', 'opportunities_prob', 'skills_prob', 'attitude_prob', 'midterm_prob', 'essay_prob', 'understanding_prob', 'books_prob', 'project_prob', 'participation_prob', 'projects_prob', 'content_prob', 'lessons_prob', 'labs_prob', 'advice_prob', 'lab_prob', 'mistakes_prob', 'credits_prob', 'teaching_prob', 'style_prob', 'discussions_prob', 'email_prob', 'criteria_prob', 'passion_prob', 'environment_prob', 'powerpoints_prob', 'homeworks_prob', 'info_prob', 'guide_prob', 'score_prob', 'expectations_prob', 'money_prob', 'articles_prob', 'respect_prob', 'presentation_prob', 'resources_prob', 'blackboard_prob', 'phone_prob', 'directions_prob', 'curve_prob', 'opinion_prob', 'discussion_prob', 'deadlines_prob', 'sessions_prob', 'powerpoint_prob', 'group_prob', 'presentations_prob', 'guidelines_prob', 'stress_prob', 'grader_prob', 'home_prob', 'tools_prob', 'response_prob', 'format_prob', 'textbooks_prob', 'schedule_prob', 'session_prob', 'note_prob', 'research_prob', 'formulas_prob', 'background_prob', 'explanations_prob', 'difficulty_prob', 'patience_prob', 'method_prob', 'terms_prob', 'terminology_prob', 'formulas_prob', 'career_prob', 'midterms_prob', 'posts_prob', 'feedbacks_prob', 'methods_prob', 'hws_prob', 'attempts_prob', 'ability_prob', 'scores_prob', 'structure_prob', 'explanation_prob', 'humor_prob', 'requirements_prob', 'practice_prob', 'policy_prob', 'tutoring_prob', 'instruction_prob', 'exercises_prob', 'classroom_prob', 'guidance_prob', 'curriculum_prob', 'deadline_prob', 'recordings_prob', 'absences_prob', 'pressure_prob', 'finals_prob', 'results_prob', 'clarification_prob']

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA

# --- Your DataFrame ---
# (make sure you load or define final_df before this)

# Specify your starting column
start_column = 'work_prob'

# Find index of start_column
start_idx = final_df.columns.get_loc(start_column)

# Select all columns after the start_column
columns_for_pca = final_df.columns[start_idx + 1:]

# Prepare data (only numeric columns, drop NaNs if needed)
X = final_df[columns_for_pca].select_dtypes(include=[np.number]).dropna()

# Check if X is empty
if X.shape[1] == 0:
    raise ValueError("No numeric columns found after the specified start column.")

# Fit PCA
pca = PCA()
X_pca = pca.fit_transform(X)

# Explained variance
explained_variance_ratio = pca.explained_variance_ratio_
cumulative_variance = np.cumsum(explained_variance_ratio)

# Plot cumulative explained variance
plt.figure(figsize=(8, 6))
plt.plot(range(1, len(cumulative_variance) + 1), cumulative_variance, marker='o')
plt.xlabel('Number of Components')
plt.ylabel('Cumulative Explained Variance')
plt.title('PCA Explained Variance')
plt.grid(True)
plt.show()

#We will choose 10 components as it describes 75% of the variance.
pca = PCA(n_components=10)
X_pca = pca.fit_transform(X)

# X_pca now has shape (number_of_samples, 10)
# Optional: Save PCA results to a DataFrame
pca_df = pd.DataFrame(X_pca, columns=[f'PC{i+1}' for i in range(10)])

# If you want to attach it back to your original DataFrame
final_df_pca = pd.concat([final_df.reset_index(drop=True), pca_df], axis=1)

pca_df.head()

# Assuming you already have:
# - pca (your fitted PCA model)
# - final_df (your original DataFrame)
# - columns_for_pca (the columns you used for PCA)

import pandas as pd

# 1. Get the loadings (feature contributions to each principal component)
loadings = pd.DataFrame(
    pca.components_.T, 
    columns=[f'PC{i+1}' for i in range(pca.n_components_)],
    index=final_df[columns_for_pca].columns
)

# 2. Function to print top features for each PC
def print_top_features_per_pc(loadings, top_n=15):
    for pc in loadings.columns:
        print(f"\nTop {top_n} features contributing to {pc}:")
        top_features = loadings[pc].sort_values(key=abs, ascending=False).head(top_n)
        print(top_features)

# 3. Run it for all 10 components
print_top_features_per_pc(loadings, top_n=15)

Top 15 features contributing to PC1:
classes_prob       -0.154978
classroom_prob     -0.140785
lecture_prob       -0.131693
homework_prob      -0.130815
teaching_prob      -0.129721
homeworks_prob     -0.128179
assignment_prob    -0.127797
assignments_prob   -0.122146
grade_prob         -0.117973
textbook_prob      -0.117791
project_prob       -0.115395
lectures_prob      -0.114446
essay_prob         -0.111184
subject_prob       -0.111027
exam_prob          -0.110518
Name: PC1, dtype: float32

Top 15 features contributing to PC2:
classes_prob        0.206274
lectures_prob       0.202470
classroom_prob      0.185897
textbooks_prob      0.181004
lecture_prob        0.177835
assignments_prob    0.177634
syllabus_prob       0.173720
curriculum_prob     0.170568
textbook_prob       0.169321
tutoring_prob       0.159937
time_prob          -0.158659
grades_prob         0.145716
exams_prob          0.136665
money_prob         -0.134526
response_prob      -0.132416
Name: PC2, dtype: float32

Top 15 features contributing to PC3:
grades_prob          -0.265772
grade_prob           -0.262167
grader_prob          -0.237692
gpa_prob             -0.198967
presentations_prob    0.189515
feedbacks_prob        0.177979
discussions_prob      0.161399
presentation_prob     0.156826
topics_prob           0.155293
attendance_prob      -0.150126
resources_prob        0.149068
feedback_prob         0.137256
score_prob           -0.135851
articles_prob         0.125053
scores_prob          -0.120262
Name: PC3, dtype: float32

Top 15 features contributing to PC4:
classes_prob     -0.278308
quizzes_prob      0.222737
quiz_prob         0.212409
lecture_prob     -0.190992
tests_prob        0.187831
scores_prob       0.176579
classroom_prob   -0.174636
readings_prob     0.173641
exam_prob         0.173066
exams_prob        0.161945
lectures_prob    -0.159325
teaching_prob    -0.157873
homeworks_prob    0.156367
test_prob         0.150748
questions_prob    0.147703
Name: PC4, dtype: float32

Top 15 features contributing to PC5:
feedback_prob    -0.264975
feedbacks_prob   -0.262013
materials_prob    0.232290
material_prob     0.190682
formulas_prob     0.186051
formulas_prob     0.186051
formulas_prob     0.186051
formulas_prob     0.186051
textbook_prob     0.156209
paper_prob        0.149172
papers_prob       0.146416
grades_prob      -0.144616
books_prob        0.140906
opinion_prob     -0.139617
attention_prob   -0.139009
Name: PC5, dtype: float32

Top 15 features contributing to PC6:
powerpoint_prob       0.238069
powerpoints_prob      0.219425
assignments_prob      0.176499
deadlines_prob        0.175353
effort_prob           0.173785
questions_prob       -0.171767
projects_prob         0.167357
books_prob           -0.166646
quizzes_prob         -0.161814
presentations_prob    0.161266
assignment_prob       0.160318
textbooks_prob       -0.158858
deadline_prob         0.158758
presentation_prob     0.157897
essays_prob           0.155557
Name: PC6, dtype: float32

Top 15 features contributing to PC7:
slides_prob           0.258428
presentation_prob     0.208597
presentations_prob    0.200320
lectures_prob         0.184220
career_prob          -0.171681
session_prob          0.168346
grades_prob          -0.165873
lecture_prob          0.162015
essay_prob           -0.159524
advice_prob          -0.158352
powerpoint_prob       0.140035
group_prob            0.139301
sessions_prob         0.136278
essays_prob          -0.127092
curve_prob            0.124276
Name: PC7, dtype: float32

Top 15 features contributing to PC8:
classes_prob         -0.241347
deadlines_prob        0.213755
opportunities_prob    0.188994
deadline_prob         0.188463
schedule_prob         0.187276
sessions_prob         0.178184
materials_prob       -0.177697
structure_prob       -0.171268
material_prob        -0.167202
method_prob          -0.157826
attendance_prob       0.152589
style_prob           -0.135841
grade_prob           -0.134656
time_prob             0.129197
topics_prob          -0.124073
Name: PC8, dtype: float32

Top 15 features contributing to PC9:
questions_prob       -0.232051
topics_prob          -0.201380
quiz_prob            -0.194867
guide_prob            0.187592
instructions_prob     0.185674
feedback_prob         0.174618
directions_prob       0.165549
instruction_prob      0.159985
feedbacks_prob        0.151607
participation_prob   -0.144752
opportunities_prob   -0.140299
textbook_prob         0.139378
lessons_prob          0.133581
guidance_prob         0.133565
quizzes_prob         -0.133225
Name: PC9, dtype: float32

Top 15 features contributing to PC10:
credit_prob         -0.223183
credits_prob        -0.215186
notes_prob          -0.211415
method_prob          0.204563
methods_prob         0.195886
practice_prob        0.180541
papers_prob         -0.179508
note_prob           -0.168688
directions_prob      0.168071
difficulty_prob      0.163215
tutoring_prob        0.160847
instruction_prob     0.158615
reading_prob        -0.157121
instructions_prob    0.152964
paper_prob          -0.150175
Name: PC10, dtype: float32

from sklearn.cluster import KMeans
import matplotlib.pyplot as plt

# Try different number of clusters
inertia = []
K = range(1, 10)  # Test k from 1 to 10

for k in K:
    kmeans = KMeans(n_clusters=k, random_state=42)
    kmeans.fit(X_pca)
    inertia.append(kmeans.inertia_)

# Plot
plt.figure(figsize=(8, 6))
plt.plot(K, inertia, marker='o')
plt.xlabel('Number of clusters (k)')
plt.ylabel('Inertia')
plt.title('Elbow Method for Optimal k')
plt.grid(True)
plt.show()

/usr/local/lib/python3.11/dist-packages/sklearn/cluster/_kmeans.py:870: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/sklearn/cluster/_kmeans.py:870: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/sklearn/cluster/_kmeans.py:870: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/sklearn/cluster/_kmeans.py:870: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/sklearn/cluster/_kmeans.py:870: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/sklearn/cluster/_kmeans.py:870: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/sklearn/cluster/_kmeans.py:870: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/sklearn/cluster/_kmeans.py:870: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/sklearn/cluster/_kmeans.py:870: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  warnings.warn(

from sklearn.cluster import KMeans

# Define number of clusters
n_clusters = 4

# Fit KMeans on your 10 PCA components
kmeans = KMeans(n_clusters=n_clusters, random_state=42)
cluster_labels = kmeans.fit_predict(X_pca)

# Optional: Create a DataFrame for cluster results
pca_df['cluster'] = cluster_labels  # If you have a pca_df
# OR
# final_df_pca['cluster'] = cluster_labels  # If you attached PCA to original df

# Quick check: number of points per cluster
print(np.bincount(cluster_labels))

/usr/local/lib/python3.11/dist-packages/sklearn/cluster/_kmeans.py:870: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  warnings.warn(

[81470 68755 50546 36636]

pca_df

import pandas as pd
import numpy as np

# If you want to work with your PCA components
pca_df = pd.DataFrame(X_pca, columns=[f'PC{i+1}' for i in range(10)])
pca_df['cluster'] = cluster_labels

# Calculate mean for each cluster
cluster_summary = pca_df.groupby('cluster').mean()

print(cluster_summary)

              PC1       PC2       PC3       PC4       PC5       PC6       PC7  \
cluster                                                                         
0       -0.077084  0.451393  0.011973  0.002916 -0.013079 -0.000555 -0.017195   
1        0.036293 -0.542030 -0.048198 -0.005592  0.034939 -0.032973  0.032487   
2        0.825189  0.029089  0.048521  0.002016 -0.020884  0.030184 -0.011944   
3       -1.035192 -0.026696 -0.003114  0.001229 -0.007674  0.021470 -0.006251   

              PC8       PC9      PC10  
cluster                                
0        0.000318  0.007642  0.004058  
1        0.014085  0.010353 -0.003490  
2       -0.010736 -0.015221  0.000165  
3       -0.012330 -0.015425 -0.002703

# Step 1: Drop all columns after 'objects_only'
cutoff_index = final_df.columns.get_loc('objects_only') + 1
final_df_trimmed = final_df.iloc[:, :cutoff_index]

# Step 2: Drop 'Unnamed: 0' column if it exists
if 'Unnamed: 0' in final_df_trimmed.columns:
    final_df_trimmed = final_df_trimmed.drop(columns=['Unnamed: 0'])

# Step 3: Add the cluster labels from pca_df
final_df_trimmed['cluster'] = pca_df['cluster'].values  # Ensure matching index

# Final result
final_df = final_df_trimmed

final_df.to_csv('final_df_with_clusters.csv', index=False)
cluster_df=final_df
final_df

cluster_df=pd.read_csv("final_df_with_clusters.csv")

cluster_df

sentiment_map = {'NEGATIVE': 0, 'POSITIVE': 1}

cluster_df['sentiment_numeric'] = cluster_df['sentiment_label_transformer'].map(sentiment_map)

# Calculate total reviews and average sentiment per college
summary = cluster_df.groupby('College').agg(
    total_reviews=('sentiment_numeric', 'count'),
    avg_sentiment_score=('sentiment_numeric', 'mean')
).reset_index()

# Filter for colleges with more than 1000 reviews
summary_filtered = summary[summary['total_reviews'] > 1000].sort_values(by='avg_sentiment_score', ascending=False)

# Display the result
print(summary_filtered)

                                            College  total_reviews  \
14                         Hostos Community College           5785   
2                           Bronx Community College           6784   
16                   Kingsborough Community College          10582   
17                      LaGuardia Community College          16149   
1            Borough of Manhattan Community College          24119   
19                             Medgar Evers College           3853   
18                                   Lehman College          12371   
12  College of Staten Island (CUNY - Staten Island)          13506   
21                  Queensborough Community College          12336   
10                                CUNY York College           6562   
15             John Jay College of Criminal Justice          20093   
11                         City College of New York          17045   
20              New York City College of Technology          12643   
5                               CUNY Queens College          27435   
3                                  Brooklyn College          15004   
0                                    Baruch College          31610   
8               CUNY School of Professional Studies           1082   

    avg_sentiment_score  
14             0.769231  
2              0.760613  
16             0.739936  
17             0.712366  
1              0.709897  
19             0.680768  
18             0.679977  
12             0.674811  
21             0.672503  
10             0.671746  
15             0.660728  
11             0.648929  
20             0.640433  
5              0.638272  
3              0.634031  
0              0.631572  
8              0.597967

import matplotlib.pyplot as plt
import seaborn as sns
cluster_df['sentiment_numeric'] = cluster_df['sentiment_label_transformer'].map(sentiment_map)

# Count number of reviews per college
college_counts = cluster_df['College'].value_counts()

# Filter colleges with more than 1000 reviews
valid_colleges = college_counts[college_counts > 1000].index
filtered_df = cluster_df[cluster_df['College'].isin(valid_colleges)]

# Calculate average sentiment score per college
avg_sentiment = filtered_df.groupby('College')['sentiment_numeric'].mean().sort_values()

# Convert to DataFrame for heatmap
avg_sentiment_df = avg_sentiment.to_frame().T  # Transpose so colleges are columns

# Plot heatmap
plt.figure(figsize=(15, 2))  # Wide horizontal heatmap
sns.heatmap(avg_sentiment_df, annot=True, cmap='coolwarm_r', cbar_kws={'label': 'Average Sentiment Score'})
plt.title('Average Sentiment Score by College (Only Colleges with >1000 Reviews)')
plt.yticks([])  # Hide y-axis labels
plt.xticks(rotation=45, ha='right')  # Rotate x-axis labels
plt.tight_layout()
plt.show()

/tmp/ipykernel_31/3026761609.py:24: UserWarning: Tight layout not applied. The bottom and top margins cannot be made large enough to accommodate all axes decorations.
  plt.tight_layout()

cluster_df['sentiment_numeric'] = cluster_df['sentiment_label_transformer'].map(sentiment_map)

# Compute averages per college
college_stats = cluster_df.groupby('College').agg(
    total_reviews=('sentiment_numeric', 'count'),
    avg_sentiment=('sentiment_numeric', 'mean'),
    avg_difficulty=('Difficulty', 'mean'),
    avg_quality=('Quality', 'mean')
).reset_index()

# Filter for colleges with more than 1000 reviews
filtered_stats = college_stats[college_stats['total_reviews'] > 1000]

# Compute correlations
correlations = filtered_stats[['avg_sentiment', 'avg_difficulty', 'avg_quality']].corr()

# Display results
print("College Averages:")
print(filtered_stats.sort_values('avg_sentiment', ascending=False))
print("\nCorrelations:")
print(correlations)

College Averages:
                                            College  total_reviews  \
14                         Hostos Community College           5785   
2                           Bronx Community College           6784   
16                   Kingsborough Community College          10582   
17                      LaGuardia Community College          16149   
1            Borough of Manhattan Community College          24119   
19                             Medgar Evers College           3853   
18                                   Lehman College          12371   
12  College of Staten Island (CUNY - Staten Island)          13506   
21                  Queensborough Community College          12336   
10                                CUNY York College           6562   
15             John Jay College of Criminal Justice          20093   
11                         City College of New York          17045   
20              New York City College of Technology          12643   
5                               CUNY Queens College          27435   
3                                  Brooklyn College          15004   
0                                    Baruch College          31610   
8               CUNY School of Professional Studies           1082   

    avg_sentiment  avg_difficulty  avg_quality  
14       0.769231        2.968885     4.045549  
2        0.760613        2.852300     4.063016  
16       0.739936        2.862880     4.006095  
17       0.712366        3.034739     3.866741  
1        0.709897        2.997015     3.871408  
19       0.680768        3.076564     3.712821  
18       0.679977        3.097001     3.791448  
12       0.674811        3.041537     3.749334  
21       0.672503        2.972033     3.750892  
10       0.671746        3.087931     3.732703  
15       0.660728        2.882646     3.775146  
11       0.648929        3.152361     3.685186  
20       0.640433        3.079491     3.635332  
5        0.638272        3.173574     3.627738  
3        0.634031        3.149294     3.648660  
0        0.631572        3.096299     3.684499  
8        0.597967        3.287431     3.556377  

Correlations:
                avg_sentiment  avg_difficulty  avg_quality
avg_sentiment        1.000000       -0.778975     0.976398
avg_difficulty      -0.778975        1.000000    -0.824613
avg_quality          0.976398       -0.824613     1.000000

# Count reviews per college
college_counts = cluster_df['College'].value_counts()

# Filter colleges with more than 1000 reviews
valid_colleges = college_counts[college_counts > 1000].index
filtered_df = cluster_df[cluster_df['College'].isin(valid_colleges)]

# Cross-tab: cluster distribution per college as percentages
cluster_distribution = pd.crosstab(filtered_df['College'], filtered_df['cluster'], normalize='index') * 100

# Round to 2 decimal points
cluster_distribution = cluster_distribution.round(2)

print(cluster_distribution)

cluster                                              0      1      2      3
College                                                                    
Baruch College                                   33.32  28.58  23.19  14.91
Borough of Manhattan Community College           36.32  27.66  20.33  15.68
Bronx Community College                          34.51  32.31  16.54  16.64
Brooklyn College                                 33.66  29.64  21.17  15.53
CUNY Queens College                              34.19  28.62  22.44  14.75
CUNY School of Professional Studies              32.26  28.10  24.40  15.25
CUNY York College                                35.45  27.57  21.72  15.27
City College of New York                         33.70  28.68  23.20  14.41
College of Staten Island (CUNY - Staten Island)  33.45  29.75  21.75  15.05
Hostos Community College                         32.55  32.05  18.50  16.91
John Jay College of Criminal Justice             34.99  27.87  21.62  15.52
Kingsborough Community College                   33.50  30.71  18.99  16.80
LaGuardia Community College                      34.69  29.72  19.34  16.24
Lehman College                                   36.04  27.22  21.32  15.41
Medgar Evers College                             33.27  30.50  18.76  17.47
New York City College of Technology              33.24  29.95  21.15  15.67
Queensborough Community College                  35.19  28.66  21.13  15.01

import matplotlib.pyplot as plt
import seaborn as sns

plt.figure(figsize=(12, 6))
sns.heatmap(cluster_distribution, annot=True, fmt=".2f", cmap="Blues", cbar_kws={'label': '% of Comments'})
plt.title("Cluster Distribution per College (Only Colleges with >1000 Reviews)")
plt.xlabel("Cluster")
plt.ylabel("College")
plt.xticks(rotation=0)
plt.yticks(rotation=0)
plt.tight_layout()
plt.show()

cluster_df['sentiment_numeric'] = cluster_df['sentiment_label_transformer'].map({'NEGATIVE': 0, 'POSITIVE': 1})

cluster_metrics = cluster_df.groupby('cluster')[['Quality', 'Difficulty', 'sentiment_numeric']].mean().sort_index()
cluster_metrics.rename(columns={'sentiment_numeric': 'avg_sentiment'}, inplace=True)

print(cluster_metrics)

          Quality  Difficulty  avg_sentiment
cluster                                     
0        3.748779    3.073033       0.665435
1        3.839852    2.990866       0.699949
2        3.567206    3.159696       0.614905
3        3.899511    2.961513       0.705754

import pandas as pd
final_df = pd.read_csv("student_reviews_with_objects.csv")

import pandas as pd
import numpy as np
import torch
from sentence_transformers import SentenceTransformer, util
from tqdm import tqdm


labels = ['work','questions','grade','material','time','attention','notes','feedback','assignments','homework','course','lectures','credit',
 'classes','textbook','exams','quizzes','grades','slides','effort','book','papers','readings','help','tests','paper','topics','lecture',
 'hw','fun','test','essays','gpa','exam','emails','assignment','information','examples','videos','subject','quiz','knowledge','instructions',
 'answers','syllabus','attendance','chapters','opportunity','materials','accent','reading','final','opportunities','skills','attitude','midterm',
 'essay','understanding','books','project','participation','projects','content','lessons','labs','advice','lab','mistakes','credits','teaching','style',
 'discussions','email','criteria','passion','environment','powerpoints','homeworks','info','guide','score','expectations','money','articles','respect',
 'presentation','resources','blackboard','phone','directions','curve','opinion','discussion','deadlines','sessions','powerpoint','group','presentations',
 'guidelines','stress','grader','home','tools','response','format','textbooks','schedule','session','note','research','formulas','background',
 'explanations','difficulty','patience','method','terms','terminology','formulas','career','midterms','posts','feedbacks','methods','hws','attempts',
 'ability','scores','structure','explanation','humor','requirements','practice','policy','tutoring','instruction','exercises','classroom','guidance',
 'curriculum','deadline','recordings','absences','pressure','finals','results','clarification']

# Load model and put it on GPU
device = 'cuda' if torch.cuda.is_available() else 'cpu'
model = SentenceTransformer('all-MiniLM-L6-v2', device=device)

# Encode labels once (on GPU)
label_embeddings = model.encode(labels, convert_to_tensor=True, device=device)

# Batch size (adjust depending on your GPU memory)
batch_size = 64  

# Prepare list to collect results
similarities_list = []

# Process comments in batches
for i in tqdm(range(0, len(final_df), batch_size), desc="Processing Batches"):
    batch_comments = final_df['Comment'].iloc[i:i+batch_size].tolist()

    # Encode batch (on GPU)
    comment_embeddings = model.encode(batch_comments, convert_to_tensor=True, device=device)

    # Compute cosine similarities (on GPU)
    cosine_scores = util.cos_sim(comment_embeddings, label_embeddings)

    # Move to CPU and convert to numpy
    batch_similarities = cosine_scores.cpu().numpy()

    similarities_list.append(batch_similarities)

# Stack all batches
all_similarities = np.vstack(similarities_list)

# Create DataFrame
similarities_df = pd.DataFrame(all_similarities, columns=[f"{label}_prob" for label in labels])

# Final merged DataFrame
final_df = pd.concat([final_df.reset_index(drop=True), similarities_df], axis=1)

# Done!
#final_df.to_csv('df_with_labels.csv', index=False)

Batches:   0%|          | 0/5 [00:00<?, ?it/s]

Processing Batches:   0%|          | 0/3710 [00:00<?, ?it/s]

Batches:   0%|          | 0/2 [00:00<?, ?it/s]

Batches:   0%|          | 0/2 [00:00<?, ?it/s]

Batches:   0%|          | 0/2 [00:00<?, ?it/s]

Processing Batches:   0%|          | 3/3710 [00:00<02:50, 21.71it/s]

Batches:   0%|          | 0/2 [00:00<?, ?it/s]

Batches:   0%|          | 0/2 [00:00<?, ?it/s]

Batches:   0%|          | 0/2 [00:00<?, ?it/s]

Processing Batches:   0%|          | 6/3710 [00:00<02:45, 22.36it/s]

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA

# --- Your DataFrame ---
# (make sure you load or define final_df before this)

# Specify your starting column
start_column = 'work_prob'

# Find index of start_column
start_idx = final_df.columns.get_loc(start_column)

# Select all columns after the start_column
columns_for_pca = final_df.columns[start_idx + 1:]

# Prepare data (only numeric columns, drop NaNs if needed)
X = final_df[columns_for_pca].select_dtypes(include=[np.number]).dropna()

# Check if X is empty
if X.shape[1] == 0:
    raise ValueError("No numeric columns found after the specified start column.")

# Fit PCA
pca = PCA()
X_pca = pca.fit_transform(X)

# Explained variance
explained_variance_ratio = pca.explained_variance_ratio_
cumulative_variance = np.cumsum(explained_variance_ratio)

# Plot cumulative explained variance
plt.figure(figsize=(8, 6))
plt.plot(range(1, len(cumulative_variance) + 1), cumulative_variance, marker='o')
plt.xlabel('Number of Components')
plt.ylabel('Cumulative Explained Variance')
plt.title('PCA Explained Variance')
plt.grid(True)
plt.show()

#We will choose 10 components as it describes 75% of the variance.
pca = PCA(n_components=10)
X_pca = pca.fit_transform(X)

# X_pca now has shape (number_of_samples, 20)
# Optional: Save PCA results to a DataFrame
pca_df = pd.DataFrame(X_pca, columns=[f'PC{i+1}' for i in range(10)])

# If you want to attach it back to your original DataFrame
final_df_pca = pd.concat([final_df.reset_index(drop=True), pca_df], axis=1)

pca_df.head()

import pandas as pd

# 1. Get the loadings (feature contributions to each principal component)
loadings = pd.DataFrame(
    pca.components_.T, 
    columns=[f'PC{i+1}' for i in range(pca.n_components_)],
    index=final_df[columns_for_pca].columns
)

# 2. Function to print top features for each PC
def print_top_features_per_pc(loadings, top_n=20):
    for pc in loadings.columns:
        print(f"\nTop {top_n} features contributing to {pc}:")
        top_features = loadings[pc].sort_values(key=abs, ascending=False).head(top_n)
        print(top_features)

# 3. Run it for all 20 components
print_top_features_per_pc(loadings, top_n=10)

Top 10 features contributing to PC1:
homeworks_prob      0.145388
quizzes_prob        0.138905
textbook_prob       0.130173
topics_prob         0.126009
exam_prob           0.123768
lectures_prob       0.122766
instruction_prob    0.122722
books_prob          0.120268
resources_prob      0.120228
homework_prob       0.119782
Name: PC1, dtype: float32

Top 10 features contributing to PC2:
tutoring_prob    -0.206863
lectures_prob    -0.198196
exams_prob       -0.185684
lecture_prob     -0.179379
textbooks_prob   -0.173588
syllabus_prob    -0.167377
textbook_prob    -0.165699
classes_prob     -0.163064
grades_prob      -0.161355
classroom_prob   -0.157187
Name: PC2, dtype: float32

Top 10 features contributing to PC3:
exams_prob       0.221748
exam_prob        0.216943
grades_prob      0.214366
tests_prob       0.204830
test_prob        0.200993
lectures_prob   -0.192415
scores_prob      0.187033
grade_prob       0.182502
score_prob       0.179274
lecture_prob    -0.161719
Name: PC3, dtype: float32

Top 10 features contributing to PC4:
grader_prob         -0.242163
classroom_prob      -0.240828
classes_prob        -0.230124
grades_prob         -0.221930
grade_prob          -0.212572
attendance_prob     -0.198566
directions_prob      0.194602
instructions_prob    0.188238
teaching_prob       -0.180682
guide_prob           0.167086
Name: PC4, dtype: float32

Top 10 features contributing to PC5:
quizzes_prob        0.227477
essay_prob         -0.223752
lectures_prob       0.221110
essays_prob        -0.195119
tests_prob          0.193090
sessions_prob       0.180346
test_prob           0.166261
lecture_prob        0.165301
assignments_prob   -0.165170
exam_prob           0.163449
Name: PC5, dtype: float32

Top 10 features contributing to PC6:
labs_prob            -0.272904
lab_prob             -0.231421
books_prob           -0.213782
textbooks_prob       -0.191792
presentations_prob    0.182396
presentation_prob     0.181233
readings_prob        -0.163235
effort_prob           0.151403
research_prob        -0.148849
career_prob          -0.148510
Name: PC6, dtype: float32

Top 10 features contributing to PC7:
schedule_prob         0.237718
attendance_prob       0.236130
explanations_prob    -0.217977
requirements_prob     0.205895
understanding_prob   -0.190459
explanation_prob     -0.188764
teaching_prob        -0.179123
deadline_prob         0.174043
participation_prob    0.165606
deadlines_prob        0.165598
Name: PC7, dtype: float32

Top 10 features contributing to PC8:
essays_prob        -0.250903
skills_prob         0.220026
essay_prob         -0.216735
practice_prob       0.212846
tutoring_prob       0.191946
projects_prob       0.189581
career_prob         0.188591
discussions_prob   -0.159441
teaching_prob       0.158419
discussion_prob    -0.149532
Name: PC8, dtype: float32

Top 10 features contributing to PC9:
lab_prob            0.247814
formulas_prob       0.230438
formulas_prob       0.230438
formulas_prob       0.230438
formulas_prob       0.230438
labs_prob           0.202996
slides_prob         0.182226
powerpoints_prob    0.181821
reading_prob       -0.172788
powerpoint_prob     0.153661
Name: PC9, dtype: float32

Top 10 features contributing to PC10:
powerpoint_prob       0.277634
powerpoints_prob      0.272390
slides_prob           0.195905
presentations_prob    0.185703
presentation_prob     0.163720
guidance_prob        -0.157086
feedbacks_prob        0.155008
feedback_prob         0.150017
practice_prob        -0.146455
terms_prob           -0.144140
Name: PC10, dtype: float32

from sklearn.cluster import KMeans
import matplotlib.pyplot as plt

# Try different number of clusters
inertia = []
K = range(1, 10)  # Test k from 1 to 10

for k in K:
    kmeans = KMeans(n_clusters=k, random_state=42)
    kmeans.fit(X_pca)
    inertia.append(kmeans.inertia_)

# Plot
plt.figure(figsize=(8, 6))
plt.plot(K, inertia, marker='o')
plt.xlabel('Number of clusters (k)')
plt.ylabel('Inertia')
plt.title('Elbow Method for Optimal k')
plt.grid(True)
plt.show()

/usr/local/lib/python3.11/dist-packages/sklearn/cluster/_kmeans.py:870: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/sklearn/cluster/_kmeans.py:870: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/sklearn/cluster/_kmeans.py:870: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/sklearn/cluster/_kmeans.py:870: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/sklearn/cluster/_kmeans.py:870: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/sklearn/cluster/_kmeans.py:870: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/sklearn/cluster/_kmeans.py:870: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/sklearn/cluster/_kmeans.py:870: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/sklearn/cluster/_kmeans.py:870: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  warnings.warn(

from sklearn.cluster import KMeans

# Define number of clusters
n_clusters = 3

# Fit KMeans on your 10 PCA components
kmeans = KMeans(n_clusters=n_clusters, random_state=42)
cluster_labels = kmeans.fit_predict(X_pca)

# Optional: Create a DataFrame for cluster results
pca_df['cluster'] = cluster_labels  # If you have a pca_df
# OR
# final_df_pca['cluster'] = cluster_labels  # If you attached PCA to original df

# Quick check: number of points per cluster
print(np.bincount(cluster_labels))

/usr/local/lib/python3.11/dist-packages/sklearn/cluster/_kmeans.py:870: FutureWarning: The default value of `n_init` will change from 10 to 'auto' in 1.4. Set the value of `n_init` explicitly to suppress the warning
  warnings.warn(

[ 62098  61538 113771]

import pandas as pd
import numpy as np

# If you want to work with your PCA components
pca_df = pd.DataFrame(X_pca, columns=[f'PC{i+1}' for i in range(10)])
pca_df['cluster'] = cluster_labels

# Calculate mean for each cluster
cluster_summary = pca_df.groupby('cluster').mean()

print(cluster_summary)

              PC1       PC2       PC3       PC4       PC5       PC6       PC7  \
cluster                                                                         
0       -0.591839  0.049099  0.013089 -0.009288 -0.000791  0.011894 -0.001157   
1        0.593978  0.045060  0.012219  0.003914 -0.002289  0.014389  0.003050   
2        0.001756 -0.051172 -0.013754  0.002952  0.001670 -0.014275 -0.001018   

              PC8       PC9      PC10  
cluster                                
0        0.001306 -0.000344 -0.000703  
1       -0.000807 -0.000025 -0.001035  
2       -0.000276  0.000201  0.000943

	PC1	PC2	PC3	PC4	PC5	PC6	PC7	PC8	PC9	PC10
0	0.245638	0.180714	-0.221023	0.322108	-0.165037	-0.092894	-0.007801	-0.081985	0.007758	0.287692
1	-1.436934	-0.361553	-0.057584	0.763694	-0.228414	-0.302517	0.127172	0.222630	-0.213423	0.259219
2	0.514291	0.709100	0.057545	0.511209	-0.174763	-0.031684	-0.113555	0.101319	-0.098681	0.282336
3	0.439777	0.829811	-0.046056	0.462712	0.053805	0.003133	-0.018555	0.100676	-0.173517	0.366785
4	0.079189	0.234170	-0.020774	-0.118356	-0.329299	0.175446	-0.098058	0.096107	0.201906	-0.152528

	PC1	PC2	PC3	PC4	PC5	PC6	PC7	PC8	PC9	PC10	cluster
0	0.245638	0.180714	-0.221023	0.322108	-0.165037	-0.092894	-0.007801	-0.081985	0.007758	0.287692	0
1	-1.436934	-0.361553	-0.057584	0.763694	-0.228414	-0.302517	0.127172	0.222630	-0.213423	0.259219	3
2	0.514291	0.709100	0.057545	0.511209	-0.174763	-0.031684	-0.113555	0.101319	-0.098681	0.282336	0
3	0.439777	0.829811	-0.046056	0.462712	0.053805	0.003133	-0.018555	0.100676	-0.173517	0.366785	0
4	0.079189	0.234170	-0.020774	-0.118356	-0.329299	0.175446	-0.098058	0.096107	0.201906	-0.152528	0
...	...	...	...	...	...	...	...	...	...	...	...
237402	-0.293803	-0.063315	-0.161420	0.220347	-0.168907	0.222322	-0.261293	0.018199	0.101693	-0.017203	0
237403	0.356559	0.953208	0.025442	0.138864	0.294135	0.016914	-0.182517	0.117201	-0.064719	0.000135	0
237404	0.143879	-0.520007	-0.120729	0.001714	0.000043	-0.007120	0.018714	-0.108075	0.040783	-0.047760	1
237405	0.116618	-0.650743	0.268947	0.072335	0.093666	0.115173	-0.081280	0.024438	-0.056108	0.267761	1
237406	-0.377857	-0.206008	-0.310988	0.378781	0.151291	-0.112730	0.097784	0.074770	0.000110	0.051972	1

	PC1	PC2	PC3	PC4	PC5	PC6	PC7	PC8	PC9	PC10
0	-0.220479	-0.091875	0.316833	-0.051618	-0.058918	0.179161	-0.152840	0.166463	0.066114	0.069672
1	0.691112	-0.111693	0.206471	-0.080795	-0.017198	0.026949	-0.007518	0.199412	0.065915	0.016454
2	0.304929	0.115624	0.260229	0.083505	0.051744	0.081797	-0.093847	0.107122	0.049310	0.231794
3	0.327226	0.011078	0.225644	-0.005435	0.032289	0.088180	-0.097384	0.169312	0.046484	0.215482
4	0.038164	-0.115254	0.040353	-0.100589	-0.054309	0.112195	0.076108	-0.056310	0.029117	0.005458

Installing Libraries¶

Import first dataset.¶

Data Cleaning¶

For the case of our project we excluded comments that:¶

Sentiment Analysis¶

Word Frequency¶

Word Frequency after reducing noise from text¶

Direct Clustering on reduced text¶

Semi-Unsupervised Learning¶

Part 1 - PCA and Clustering using the reduced text¶

Cluster interpretation:¶

Part 1b- Analysis¶

Part 2- PCA and Clustering using the original text¶

	Department	Quality	Difficulty	Comment	Date	College
0	Accounting	5.0	2	She is my recitation instructor. I have her a...	Jun 24th, 2010	Baruch College
1	Accounting	4.0	4	Easy grader for HW's. Hard exams. Must use tex...	Dec 14th, 2009	Baruch College
2	Accounting	3.5	4	had her for honors, shes not bad at all.. i th...	Nov 20th, 2009	Baruch College
3	Accounting	3.0	5	mean grader. if ur not smart,dont take her. sh...	Jun 1st, 2009	Baruch College
4	Accounting	3.0	3	Although she is pretty tough, she does want he...	May 29th, 2009	Baruch College

	Department	Quality	Difficulty	Thumbs Down	Comment	Date	College	total_votes	thumbs_down_ratio
0	Communication Studies	2.0	3	0	her class wasn't too hard. she's is just very ...	2024-10-16	Baruch College	0	0.0
1	Communication Studies	3.0	2	0	Professor Merking is overall a nice professor ...	2023-09-26	Baruch College	0	0.0
2	Communication Studies	3.0	2	0	She is alright. No hw, she gives 3 tests which...	2023-05-16	Baruch College	0	0.0
3	Communication Studies	3.0	2	0	She is alright. No hw, she gives 3 tests which...	2023-05-16	Baruch College	0	0.0
4	Communication Studies	5.0	3	1	Prof Merkin is amazing! Those who complain abo...	2023-03-30	Baruch College	1	1.0